CONFTGU^^G COMPUTER NETWORK COMPLICATIONS 

IN RESPONSE TO DETECTED FIREWALLS 

Cross Reference to Related Application 



The present application is a continuation-in-part of pending U.S. application 
serial number 09/978,616, filed October 16, 2001, entitled "Video Telephony' 1 . 



Background of the Invention 



The present invention relates to establishing a communication session 
between users connected to a computer network, and more specifically, to detecting 
the presence of firewalls connecting each user to the network and configuring the 
communication session so that network packets can be exchanged through whatever 
firewalls are present. 

t: 

Internetworking (i.e., the interconnection of many computer networks) 
allows the interaction of very large numbers of computers and computer users. The 
most well known example is the Internet. Computers connected to the Internet may be 
widely separated geographically and utilize many different hardware and software 
configurations. In order to achieve communication sessions between any two 
endpoints on the Internet, an addressing system and various standard protocols for 
exchanging computer data packets have been developed. 

Each packet sent over the Internet includes fields that specify the source and 
destination address of the packet according to Internet Protocol (IP) addresses 
assigned to the network interface nodes involved. Currently assigned addresses 
comprise 32 bits, although future standards allow for 128 bit addresses. The 32 bit 
addresses are normally written by breaking the 32 bits into 4 groups of 8 bits each and 
writing the decimal equivalents of each group separated by periods (e.g., 
208.25.106.10). 
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Since numerical IP addresses are ; inconvenient to use and remember, a 
protocol for assigning and accessing logical names is used known as the domain name 
system (DNS). DNS servers are deployed within the Internet which perform a 
translation function between a logical domain name such as "sprint.com" and its 
numerical equivalent "208.25.106.10". After receiving an IP address back from a 
DNS server, a computer can forward data packets to the IP address and establish a 
connection or session with the remote computer. 

While the DNS system works well for hosted content (e.g., material made 
available for browsing by commercial and private entities), it is not well suited to ad 
hoc communications or exchanges of data between individuals. Hosting a website and 
registering an IP address within the DNS system is expensive and time consuming. 
Furthermore, due to an impending shortage of IP addresses and the cost for 
maintaining use of each IP address, many Internet service providers assign IP 
addresses dynamically to their individual users. In other words, when a user signs on 
to their service, they are temporarily assigned an IP address from an address pool 
assigned to their service provider. The user occupies that IP address only for their 
current session. 

Even when individual users have their own static IP addresses, and when 
other users can remember the IP address of a user with whom they would like to 
establish a connection session over the Internet (e.g., for voice or video telephony), the 
need to configure their hardware or software is too complex for many users. This is 
one reason why e-mail is such a popular and successful Internet application. A mail 
server with an easy to remember domain name acts as intermediary between two 
individual users. Using a simple application program and the recipient's account name 
on the mail server (i.e., their e-mail address), text messages and computer files can be 
exchanged. The exchange, however, does not allow the users to interact in real time. 

Parent application U.S. Serial No. 09/978,616 and copending application 
U.S. Serial No. (1793), incorporated herein by reference in its entirety, teach the use of 
a central server allowing two or more individual users to establish interactive 
connection sessions over the Internet without requiring overt knowledge of the other's 
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IP address and witho^femplicated configurations or set-ups^fech user registers with 
the central server, resulting in a database of users and their current IP addresses. A 
calling user sends a request to the central server to establish a connection with a called 
user. The central server can either relay all network message packets between the 
users for the duration of a "call", or it may provide the IP addresses to the users so that 
they can exchange packets directly. 

To reduce processing load and the corresponding size of the central server, 
the provider of the central server may find it preferable to provide the IP addresses to 
the calling and/or called users so that it does not have to act as intermediary for all 
packet exchanges (e.g., receiving each packet, detecting sources and intended 
destinations, and rewriting each packet header). Handing off the connection, however, 
may be impeded if the existing sessions include any firewalls. 

Many different types of firewalls have been developed to block certain 
types of communication through the firewall. Blocking of particular packets within 
user traffic directed at the firewall can be performed based on several different criteria, 

such as IP address where the traffic originated, domain names of the source or 
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destination of the traffic, the protocol in which the traffic is formatted, and the port 
sending or receiving the traffic, among others. Firewalls can also perform proxy 
services or perform network address translation (NAT) or port address translation 
(PAT) in which a user's local (i.e., private) equipment IP address is translated into a 
global (i.e., public) IP address of the firewall, so that a particular computer is not 
directly accessible from outside the firewall. 

In the presence of firewalls, some users may only be able to participate in a 
connection session that they initiate. Thus, a calling user may not be able to get any 
response to packets it sends to an IP address that it received from the central server. If 
a firewall is performing address translation, then the IP address reported by the central 
server is the global address of the firewall and not the local equipment address of the 
user. Thus, while the user behind the firewall will continue to communicate with the 
central server (since the user initiated that session when it signed on or registered with 



the central server), the user will not communicate with a calling user who sends a 
packet to the global address of the firewall. 



Summary of the Invention 

The present invention has the advantage of detecting the presence of 
firewalls for each user of the service and dynamically adjusting the call characteristics 
to enable point-to-point communication between the calling and called users whenever 
possible. 

In one important aspect of the invention, a method connects at least two 
users to exchange network packets via an internetwork. Each user is addressable 
within the internetwork at a respective global address. Some users of the internetwork 
are connected to the internetwork via a respective firewall. The method comprises 
maintaining in a central server coupled to the internetwork a database of registered 
users. The database includes respective global addresses corresponding to the 
registered users. A call request is received from a calling user to establish a 
connection to exchange network packets with a called user, at least the called user 
being a registered user. Whether or not a respective firewall is in place between the 
called user and the internetwork is detected. If a respective firewall is not in place 
between the called user and the internetwork, then the called user's respective global 
address is transmitted to the calling user, and the calling user establishes a network 
session for the connection with the called user by transmitting to the called user's 
respective global address. If a respective firewall is in place between the called user 
and the internetwork, but a respective firewall is not in place between the calling user 
and the internetwork, then the calling user's respective global address is transmitted to 
the called user, and the called user establishes a network session for the connection 
with the calling user by transmitting to the calling user's respective global address. If 
both the called user and the calling user have a firewall, all packets exchanged 
between the calling user and the called user during the connection are relayed through 
the central server. 
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Brief Description of the Drawings 

Figure 1 is a block diagram showing the interconnection of users over the 
5 Internet to the central server of the present invention. 

Figure 2 is a block diagram showing a user connection model of the present 

invention. 

Figure 3 is a flow diagram of one preferred embodiment of the invention. 
Figure 4 is a block diagram showing a first embodiment of packet flow. 
!0 Figure 5 is a block diagram showing a second embodiment of packet flow. 

Figure 6 is a block diagram showing the interconnection of users to the 
Internet through respective firewalls. 

Figure 7 is a flowchart showing dynamic control of call characteristics to 
obtain a direct point-to-point network session between a calling user and a called user 
even though a firewall may be present. 

Figure 8 is a flowchart showing detection of an address translating firewall 
associated with a registered user. 

Figure 9 illustrates a video telephony system in an example of the invention. 
Figure 10 illustrates video system operation in an example of the invention. 
Figure 1 1 illustrates a user system in an example of the invention. 
Figure 12 illustrates user system operation in an example of the invention. 
Figure 13 illustrates user system operation in an example of the invention. 
Figure 14 illustrates user system operation in an example of the invention. 
Figure 15 illustrates a user system in an example of the invention. 
25 Figure 16 illustrates a server system in an example of the invention. 

Figure 17 illustrates server system operation in an example of the invention. 
Figure 18 illustrates server system operation in an example of the invention. 
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Derailed Description of Preferred Embodiments 

Figures 1-18 and the following description depict specific examples to teach 
those skilled in the art how to make and use the best mode of the invention. For the 
5 purpose of teaching inventive principles, some conventional aspects have been 

simplified or omitted. Those skilled in the art will appreciate variations from these 
examples that fall within the scope of the invention. Those skilled in the art will 
appreciate that the features described below can be combined in various ways to form 
multiple variations of the invention. As a result, the invention is not limited to the 
10 specific examples described below, but only by the claims and their equivalents. 

Referring to Figure 1, a plurality of user computers 10, 11, and 12, and a 
central server 13 are internetworked via the Internet 14. A plurality of routers 15 
within Internet 14 direct packets between various endpoints or nodes. Computers 10 
and 1 1 are shown as being connected to Internet routers belonging to Internet Service 
15 Providers (ISP's) 16 and 17, respectively. The connections to the ISP's may be by 
dial-up, digital subscriber line (DSL), cable modem, or integrated access device 
(IAD), for example. Central server 13 and computer 12 are shown directly connected 
to a router. 

Network communication comprises data messages or packets transferred 
r 20 between separate endpoints, such as between between computers 10, 1 1, or 12 (as 

clients) and central server 13. The packet transfer is accomplished by routers 15 using 
the IP addresses contained in each packet. Central server 13 typically has a fixed IP 
address that is listed on the DNS servers accessible to each computer. Each computer 
user can easily communicate with central server 13 by supplying its logical name (e.g., 
25 www.sprint.exchange.com) which is automatically resolved by their browser into an 
IP address by consulting a DNS server. Exchanging packets between users 10, 11, and 
12 themselves cannot usually be accomplished in the same way because the users and 
their IP addresses are not listed in the DNS system. Furthermore, users 10, 11, and 12 
may not wish to allow remote access into their computers except in certain 
30 circumstances. 
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The present invention facilitates exchanging data messages between two 
individual users by providing a specialized directory or look-up within central sever 
13. As shown in Figure 2, the present invention may be used within a system that 
functions to simultaneously establish a voice telephone call between the two 
individual computer users. In certain embodiments, the voice call serves as the user 
action that initiates the computer processing to establish the computer-to-computer 
connection. In addition, the voice call provides a way to alert the called party of the 
request to establish the computer connection and then serves to enhance the interaction 
between the two users during the exchange of computer data. However, the present 
invention also provides other methods for initiating the computer processing, and a 
simultaneous voice telephone call is not necessary in the present invention. 

Regarding the embodiment with a simultaneous voice telephone call in 
Figure 2, computers 10 and 1 1 have associated telephones 18 and 19 used by the same 
| respective users. The computers and telephones may be fixed installations (e.g., in a 
15 residence or a business office) or may be mobile devices (e.g., laptop computer and 
cellular phone), as long as both are accessible to each user at the same time. The 
telephones are connected to the public switched telephone network (PSTN) 20. 
Central server 13 provides a user look-up and interconnecting service for registered 
users. For security and/or billing purposes, access to the service preferably is tied to 
20 user ID f s and passwords. A user may be given an ID and password with initial sign-up 
for the service. Each user would manually configure the telephone number that they 
want to be associated with. When the user is "on-line" (i.e., has their computer turned 
on and connected to Internet 14), their computer sends a registration message to 
central server 13 to notify it that the user is available. Central server 13 can inspect 
25 the registration message to determine the current IP address and port number at which 
the user resides for its current connection session. Alternatively, the user may 
manually configure their IP address in some circumstances. Upon registration, central 
server 13 determines whether the user has a respective firewall as will be described in 
connection with Figure 7. In any case, central server 13 contains a database of 
30 currently active, registered users. Each user entry in the database includes fields for 
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user ID, password, telephone number, and IP address (including port number), user 
status, and a firewall flag, for example. 

In the connection model of Figure 2, a user #1 dials telephone 18 to make a 
voice call to a user #2 at telephone 19. The telephone number dialed by user #1 is 
5 captured as a target telephone identifier number and sent to computer 10 being used by 
user #1. Computer 10 forwards the target telephone number to central server 13 as 
part of an access request for establishing a connection with user #2. Central server 13 
looks up the target number in its database. When it finds the target number, central 
server 13 identifies the IP address associated with user #2 and sends an initiation 
10 message to computer 1 1 being used by user #2. The initiation message identifies user 
#1 (preferably by both telephone number and user ID) and the type of data to be 
exchanged (i.e., the application program to receive the data). User #2 answers the 
telephone voice call and learns that an initiation message was sent to their computer. 
Using computer 11, user #2 can verify the calling party as user #1 and can indicate 
[15 whether they accept the computer network connection with user #1 . Once user #2 
accepts, data messages can be exchanged between application programs running on 
computers 10 and 1 1 . The application programs can be written to perform file 
transfers of various types of files, video data or frames for video telephony, or other 
real-time data or control signals. Data exchange can continue until either user 
1 20 deactivates their application program. 

The sequence of events occurring in the present embodiment is shown in 
greater detail in Figure 3, in which user #1 events are in the left column, central server 
events in the center column, and user #2 events in the right column. In step 21, user 
#1 invokes the real-time interconnection service of the present invention. This can be 
25 configured as part of the normal start-up of their computer or can result from manually 
launching a software application or client program after start-up has finished. When 
the service is invoked by user #1, a registration message is sent to the central server in 
step 22. The registration message preferably includes the user ID and password 
assigned to user #1. In addition, the application software that creates the registration 
30 message may also determine the local IP address being used by the computer and 
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includes this as data within the registration message. The registration message would 
typically also include the telephone number being used by user # 1 . In addition to the 
IP address being explicitly added to the message by the application program for user 
#1, the IP address (and port number) is typically embedded in each packet forwarded 
5 by the network and the central server preferably extracts the automatically embedded 
IP address and port number. In step 23, the central server receives the registration 
message and adds the new user to the database or updates the user status, as necessary. 

Separately, user #2 invokes the real-time interconnection service in step 24. 
User #2 sends a registration message in step 25, and the central server receives the 
10 registration message and adds user #2 to the database or updates the user status, as 
necessary. Thereafter, the central server may periodically exchange further messages 
with each registered user to keep the user status current and to maintain an open 
session with each user, for example. When a user shuts down their application 
program or their computer, an unregister message (not shown) may also be sent to the 
111 5 central server. 

During the time that user #1 is on-line, user #1 desires to exchange 
computer data with user #2. In step 27, user #1 initiates an attempt to contact user #2 

« L 

and set up the data exchange. In a preferred embodiment, user #1 may identify user #2 
by virtue of user #2 f s telephone number. This target telephone number may preferably 
be captured from the act of dialing it on user #l's telephone equipment. According to 
one example which is described in more detail below, a dedicated module may be 
connected to user #l's telephone to detect the DTMF tones while dialing and to send 
the dialed number to user #l's computer. The target telephone number for user #2 is 
included in an access request message sent to the central server in step 28. 
25 In step 30, the central server looks up the target telephone number and gets 

the IP address (and port number) associated with user #2. The initiation message is 
sent by the central server in step 31.- 

User #2 receives the initiation message in step 32. If not already running, 
the user #2 computer launches the appropriate client application for responding to the 
30 initiation message and then prompts user #2 to either accept or reject the access 
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request. If rejected, then user #2 generates a reject message in step 33 and sends it to 
the central server. In step 34, the central server forwards the reject message to user #1, 
which then terminates the data portion of the attempted communication session in step 
35 (the voice telephone call is accepted, rejected, or terminated separately). 

If user #2 accepts the attempted contact and the request for data exchange, 
then user #2 causes their computer to generate an accept message in step 36 (e.g., by 
clicking an "accept" button in an application interface) and sends it to the central 
server. In step 37, the central server determines any needed configurations for 
accomplishing the data exchange and then configures the user #1 and user #2 
endpoints in step 38. The two main configurations for the data exchange will be 
described in connection with Figures 4 and 5 and are selected on the basis of detecting 
a firewall as shown in Figures 6 and 7 and the adjustment of call characteristics as 
shown in Figure 8. The user #1 and user #2 computers accept the configuration and 
then begin to exchange the data messages or packets in step 39. Other configuration 
issues, such as the configuration of the client application programs exchanging the 
actual data messages can be handled within the access request message, then initiation 
message, the accept message, and/or other packets exchanged between the endpoints, 
for example. 

A first packet exchange configuration is shown in Figure 4 wherein central 
server 13 performs a relay function such that all packets exchanged between computer 
10 and computer 1 1 pass through central server 13. In other words, after a desired 

user (called party) accepts the data call and central server notifies the first user (calling 

■. . 

party) of the acceptance, both endpoints continue to address their sent packets to 
central server 13. At central server 13, each packet is redirected by substitution of IP 
addresses. For example, a packet sent from computer 10 including its own IP address 
as the source address of the packet and the IP address of central server 13 as the 
destination address of the packet is modified after being received by central server 13 
to have the central server's address as its source address and to have the IP address of 
computer 1 1 as its destination address. After modification, central server 13 sends the 
packet back to its router and on to computer 1 1 . The same operations are used to send 
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packets from computed 1 to computer 10. The embodimen^^Figure 4 has the 
advantage that greater privacy of a user's IP address is maintained since each user's 
computer only needs to see the IP address of central server 13. Furthermore, this 
configuration can readily function in the presence of network address translation 
5 (NAT) firewalls at the endpoints. 

Figure 5 shows an alternative configuration in which direct packet exchange 
between computers 10 and 1 1 is realized. Central server 13 provides a look-up 
function and a connection initiation function. If desired user #2 (called party) accepts 
a data call, then central server 13 provides the IP address of computer 1 1 to computer 
10 10 and provides the IP address of computer 10 to computer 11. Thereafter, each 
computer can send packets addressed to the other computer and the packets are no 
longer relayed through central server 13. This embodiment has the advantage that 
central server 13 may be reduced in size since less traffic flows through it. 

The handling of firewalls in the context of a real-time interconnection 
service will be described in greater detail with reference to Figures 6-8. As shown in 
Figure 6, computer 1 1 is connected to Internet 14 through a respective firewall 41. 
Computers 10 and 1 1 contain application programs 42 and 43 that are adapted to 
interact with central server 13 and then to exchange data messages (e.g., files, video 
frames, etc.) with other users and to display or otherwise utilize the exchanged data. 

Within the total user group that registers with central server 13, there would 
typically be a mix of users with and without firewalls. The users with firewalls 
typically will not respond to network packets they receive that are not in reply to 
network packets that they initiated. Thus, even if central server 13 provides computer 
10 with the global IP address of firewall 41, any packets it sends there will not actually 
25 reach computer 11. In this situation where the only firewall exists on the called user 
side, the present invention solves the problem by dynamically reversing the roles of 
the user's computers for establishing the network session between the two computers. 
Thus, the first packets sent directly point-to-point between the two user are initiated by 
the user with a firewall, regardless of which user is the calling user. 
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The method of the invention is shown in greater detail in Figure 7. After the 
central server receives an access request between a calling user and a called user, it 
checks to determine whether the called user has a firewall in step 50. If the called user 
does not have a firewall, then the central server sends the called user's global IP 
address to the calling user in step 5 V. In step 52, the calling user sends packets 
directly to the called user's IP address in order to establish a TCP/IP network session 
with the called user. Once the network session is established, the application 
programs can perform tasks of identification, accepting or rejecting a call, transferring 
information, displaying exchanged data, and terminating a call, for example. 

If the called user has a firewall, then the central server checks in step 53 to 
determine whether the calling user has a firewall. If not, then the central server sends 
the calling user's global IP address to the called user in step 54. In step 55, the called 
user sends packets directly to the calling user's IP address in order to establish a 
TCP/IP network session between the two users. Although the roles of called and 
15 calling users are reversed in establishing the TCP/IP network session, the original 
i roles are retained for other call aspects such as the called user's decision whether or 
| not to accept a call. 

f If it is determined in step 53 that the calling user also has a firewall, then the 

| respective network sessions between the two users and the central server are used. 

20 Thus, in step 56, the central server relays the packets between the calling and the 
called users (e.g., as shown in Figure 4). > 

Figure 8 shows the registration and firewall detection process in greater 
detail. In step 60, when a user's application program creates a registration message for 
transmission to the central server, it includes in the message the local IP address being 

25 used by the computer in its local network. In step 61, the central server receives this 
information identifying the local IP address. It also inspects the header information of 
received packets and determines the global IP address from which the message 
transfer was initiated. The central server compares the local IP address and the global 
IP address in step 62 to determine whether they match. 
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If the two Chesses do not match, then a firewall ^^esent and the central 
server stores "firewall present" data (such as a firewall flag) in step 63 as part of the 
user profile in its database. If the addresses match, then a firewall is not detected and 
the corresponding user profile is set to reflect the lack of a firewall in step 64. Even if 
the IP addresses match, it may be desirable to send a test message (e.g., using a 
different source address for the central server) in order to determine that the user will 
receive such a message. If the test message does not receive a reply, then the central 
server may instead indicate a firewall is present in step 63, for example. 

In step 65, the central server sends periodic messages to the registered user 
in order to keep the current session open between them and to update the status of the 
user. In particular, an address translating firewall will close out a session that is 
inactive for a predetermined time as short as a few minutes. The central server and/or 
the application program will exchange periodic messages to avoid the time-out, which 
would be a particular problem where a firewall is in place since the central server 
could not reestablish a session. Any TCP/IP connection session, even without any 
firewalls in place, may time-out after a predetermined time. Thus, transmission of 
periodic messages can be performed after both steps 63 and 64. 

In the event that a firewall in place for any particular user goes undetected 
by the central server, a user may have an unanticipated failure to establish a network 
session with another user. Any such failure can be reported to the central server and 
then the next lower connection strategy of Figure 7 is tried (i.e., the called user 
establishing the session if the first attempt was by calling user, or the central server 
relaying all the packets for the call). 

A video telephony system utilizing the central server will now be described 
in greater detail to provide one context in which the firewall strategy may be deployed. 

Video Telephony System Configuration and Operation - FIGS. 9-10 

Figure 9 illustrates video telephony system 100 in an example of the 
invention. Video telephony system 100 comprises user systems 101 - 104, public data 
network 1 10, server system 111, and public telephone network 120. User systems 
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101-104 communicate with one another and with server systenO 1 1 over public data 
network 1 10. User systems 101-104 also communicate with one another over public 
telephone network 120. 

Although various systems could be used within the context of the invention, 
a few exemplary systems are given for illustrative purposes. Examples of public 
telephone network 120 include local and long distance telephone companies. 
Examples of public data network 1 1 1 include Internet service providers and the 
Internet. Server system 1 1 1 could be a conventional Internet server configured with 
software to implement the invention. User systems 101-104 could include a 
conventional telephone and personal computer in addition to special purpose circuitry 
and software to implement the invention. 

Figure 10 illustrates video system 100 operation in an example of the 
invention. Public data network 1 10 is not shown for clarity, although it should be 
appreciated that communications with server system 111 occur over public data 
network 1 10. Before a video call, both users systems 101 and 104 transfer log-in 
messages to server system 111 when they are ready to initiate and/or receive video 
calls. The log-in messages generally include user names and passwords, user data 
addresses and telephone numbers, and any video call preferences. Server system 111 
checks the passwords against the user names, and if they are valid, associates each 
related user name/data address/telephone number with an indication that the user is 
ready to receive and/or initiate video calls. Server system 1 1 1 also logs any video call 
preferences. 

Subsequently, user system 101 transfers a request for a telephone call over 
public telephone network 120 to user system 104. This telephone call request could be 
as simple as picking up a telephone and dialing a telephone number. Public telephone 
network 120 transfers a telephone call request to user system 104 — typically by 
processing the dialed telephone number to ring a telephone. If user system 104 grants 
the telephone call request, such as by answering the ringing telephone, a telephone call 
is established between user systems 101 and 104 over public telephone network 120. 
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In response to the te^Lne call request, user system 101 trcfifers a video call request 
to server system 111. The video call request has the called party telephone number, 
and some caller identification information, such as user name and password, user 
telephone number and data address, and any other video call parameters. One such 
5 parameter is whether the requested video call is bidirectional or unidirectional. 
Another parameter is the type of video compression and encryption that is used. 
Server system 1 1 1 uses the called party telephone number to check if the called party 
is ready to receive video call requests, and since user system 104 has logged-in, server 
system 1 1 1 transfers a video call request to user system 104. User system 104 may 
io then present the called party with a prompt, such as an instant message or tone, to 
accept the video call request. If the called party accepts the video call request, user 
fH system 104 transfers a video call acceptance to server system 111. The acceptance 
may also have video call preferences for the called party that server system 1 1 1 
resolves against the preferences of the caller. Server system 1 1 1 transfers video call 
|15 start messages to user systems 101 and 104 indicating the resolved video call 

parameters. In response to the video call start messages, user systems 101 and 104 
generate and transfer video to server system 111. This transferred video has some 
[ indicia indicating the caller and called party, so server system 1 1 1 can associate the 
J received video with the video call. 

j 20 Server system 1 1 1 uses the user system 104 data address to transfer caller 

video to user system 104. Server system 1 1 1 uses the user system 101 data address to 
transfer called party video to user system 101. Prior to this transfer, server system 
may interwork the video to provide compatibility at the receiving end. User systems 
101 and 104 receive and display the respective video to establish the video call. 
25 Eventually, user systems 101 and 104 indicate telephone call termination to public 
telephone network 120 — typically by hanging-up their telephones. In response to 
telephone call termination, user system 101 transfers a telephone call termination 
message to server system 111. In response, server system 1 1 1 transfers a video call 
termination message to user system 104, and systems 101, 104, and 1 1 1 terminate the 
30 video call. In some cases, user system 104 may detect telephone call termination and 
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transfer a termination^^ssage to server system 1 1 1 to elimim^he need for the 
termination message from server system 111. 

In one variation to the above system, the actual video transfer may be 
directly between user systems 101 and 104 over public data network 1 10. The server 
system 1 1 1 would set-up the video call and provide the appropriate data addresses to 
user systems 101 and 104 for a peer-to-peer video transfer over public data network 
110. 



General User System Configuration and Operation - Figures 1 1-14 
10 Figure 1 1 illustrates user system 300 in an example of the invention. User 

system 300 comprises video system 301, data communication system 302, telephone 
system 303, and control system 304. Control system 304 comprises telephone 
interface 305 and data interface 306. Data communication system 302 is coupled to a 
public data communication network, and telephone interface 305 is coupled to a public 
5 telephone network. 

Video system 301 could be any system configured to generate and/or 
display video. Video system 301 may include a camera for generating video of a 
caller or called party. Video system 301 may include a television or computer monitor 
to display video. Telephone system 303 could be any system configured to initiate 
o telephone calls over a public telephone network, and could be integrated into other 
systems, such as computers, appliances, and televisions. Telephone system 303 could 
utilize wireless, wire-line, optical, or other communication media. Control system 304 
could be any system configured to initiate a video call using systems 301-302 in 
response to the Initiation of a telephone call by telephone 303. Telephone interface 
25 305 is configured to receive a called number from telephone system 303 if the 

telephone number is contemporaneously used to establish a telephone call over the 
public telephone network. In response to receiving the called telephone number, data 
interface 306 is configured to transfer the called telephone number to data 
communication system 302 for transfer to a server system over the public data 
30 network. Data system 302 could be any system configured to transfer the called 
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telephone number to^^server system over the public data r^^ork and to exchange 
video between the public data network and video system 301. 

It should be appreciated that systems 301-304 could be integrated together 
or with other systems. Various combinations of equipment could be used to 
implement user system 300. Some examples of devices that could incorporate data 
system 302 include, but are not limited to, a computer, set-top box, telephone, network 
interface card, digital assistant, information appliance, and stand-alone device. Some 
examples of devices that could incorporate control system 304 include, but are not 
limited to, a computer, telephone, modem, network interface card, set-top box, and 
stand-alone device. In addition, the functionality of data system 302 and control 
system 304 could be provided by a processing system that retrieves and executes 
software that is stored on a storage system. The storage system could comprise a disk, 
tape, integrated circuit, server, or some other memory device. 

Figures 12-13 illustrate the operation for user system 300 to initiate video 
calls in an example of the invention. User system 300 awaits an indication from the 
caller to begin initiating video calls. This indication could be an input to any of 
systems 301-304. In response to the indication, data communication system 302 
transfers a log-in message over the public data network to the server system. For 
example, data communication system 302 could be configured to automatically 
transfer the log-in message upon system start. The log-in message indicates that user 
system 300 is ready to initiate video calls, and possibly, to also receive video calls. 
The log-in message may include information such as the user name and password, user 
data address and telephone number, video call parameters, and other user data or 
registration information. 

Telephone interface 305 awaits either a telephone call from telephone 
system 303 to the public telephone network or an indication from the caller to stop 
initiating video calls. If such a stop indication is received, data interface 302 transfers 
a log-off message to the server system. For example, data communication system 302 
could be configured to automatically transfer the log-off message upon system 
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shutdown. The log-o^^message indicates that user system 3(^^not ready to initiate 
or receive video calls. 

If telephone system 303 initiates a telephone call, telephone interface 305 
obtains the called telephone number, and in response, data interface 306 transfers the 
called number to data communication system 302. Data communication system 302 
transfers a video call request to the server system. The video call request includes the 
called telephone number and other information, such as the caller user name and 
password, caller telephone number and data address, and video call parameters. The 
video call parameters indicate if the requested video call is bidirectional or 
unidirectional, and the direction if unidirectional - caller to called party or called party 
to caller. The video call parameters may also indicate requested video quality and 
security. The information in the video call request is populated by control system 304 
and/or data communication system 302. 

Data communication system 302 then awaits a video call start message. If 
the video call start message is not received, for example if the video call is unavailable 
or rejected, telephone interface 305 awaits either a telephone call or an indication from 
the caller to stop initiating video calls. If received (see Figure 13), the video call start 
message indicates the actual video call parameters for the video call. If the video call 
is bidirectional or unidirectional from caller to called party, then video system 301 
generates video of the caller that data communication system 302 transfers the caller 
to the server system. If the video call is bidirectional or unidirectional from called 
party to caller, then data communication system 302 receives video from the server 
system that video system 301 displays? Systems 301-302 would typically apply 
compression, encryption, and other video technologies to the video. 

At this point between the caller and called party, a telephone call exists over 
the public telephone network and a video call exists over the public data network. If 
the caller terminates the telephone call, such as by hanging up the telephone, telephone 
interface 305 determines that the telephone call has been terminated, and as a result, 
data interface 306 indicates the telephone call termination to data communication 
system 302. Data communication system 302 then transfers a video call termination 
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message to the server^^tem and video generation, transfer, ^fept, and display are 
terminated by user system 300. The video call is similarly terminated if a video call 
termination message is received from the server system or if the caller indicates that 
the video call should be terminated. User system 300 then awaits an additional 
telephone call or an indication from the caller to stop initiating video call requests. 

Figures 13-14 illustrate the operation for user-system 300 to receive video 
calls in an example of the invention. In Figure 14, user system 300 awaits an 
indication from the user to begin receiving video call requests. This indication could 
be an input to any of systems 301-304. In response to the start indication, data 
communication system 302 transfers a log-in message over the public data network to 
the server system. For example, data communication system 302 could be configured 
to automatically transfer the log-in message upon system start. The log-in message 
indicates that user system 300 is ready to receive video call requests, and may include 
information such as the user name and password, user data address and telephone 
number, video call parameters, and other user data or registration information. 
Data interface 302 awaits a video call request from the server system or an indication 
from the user to stop receiving video call requests. If a stop indication is received, 
data interface 302 transfers a log-off message to the server system. For example, data 
communication system 302 could be configured to automatically transfer the log-off 
message upon system shut-down. 

If data communication system 304 receives a video call request, then the 

a 

user is notified of the requested video call. The notification could be given from any 
of systems 301-301 and could include screen displays, tones, or other user signals. 
Although not shown for clarity, telephone system 303 will receive a contemporaneous 
telephone call from the public telephone network. The user indicates if the telephone 
call and/or the video call is accepted. Typically, the telephone call is accepted by 
operating telephone video system 303 to answer the call. The video call may be 
accepted with an input to one of systems 301-304, such as by pressing, a DTMF key, 
pressing a button, or selecting from a screen display. User system 300 could be 
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configured to automat^lly accept or reject the video call ba^^n whether the 
corresponding telephone call, is answered. 

If the video call is not accepted, then data communication system 302 
transfers a video call rejection to the server system and awaits either a video call 
request from the server system or an indication from the user to stop receiving video 
call requests. If the video call is accepted, then data communication system 302 
transfers a video call acceptance to the server system. The video call acceptance may 
indicate accepted video call parameters. Data communication system 302 then awaits 
a video call start message indicating the actual parameters for the video call. If the 
video call start message is received, processing proceeds as indicated above for Figure 
13. 



User System Incorporating Conventional Telephone and Computer - Figure 15 
Figure 15 illustrates user system 700 in an example of the invention. 
Advantageously, user system 700 is configured for use with conventional telephones, 
personal computers, and communication services. Advantageously, this makes video 
telephony easy for a user to implement by simply adding one small device to 
conventional devices and services. 

User system 700 includes computer system 701, telephone 703, and 
interface device 704. Computer system 701 is connected to Internet link 734 that 
provides Internet service. Telephone 703 is connected to telephone link 73 1 . 
Telephone link 735 provides telephone service. It should be appreciated that links 
734-735 may share the same physical media, especially to egress the user premises. 
Computer system 701 includes user interface 711, communication interface 712, 
processing system 713, and storage system 714. User interface 711 includes video 
equipment 718. Storage system 714: stores operating software 716 and video software 
717. Interface device 704 includes RJ-11 jacks 721-722, Dual-Tone Multi-Frequency 
(DTMF) decoder 723, call sensor 724, controller 725, and computer interface 726. 
RJ-1 1 jack 721 is coupled to telephone 703 by telephone link 73 1 . RJ-1 1 jack 722 is 
coupled to telephone link 735. Computer interface 726 is coupled to communication 
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interface 712 by Universal Serial Bus (USB) 733. Alternativel^USB 733 could be 
serial cable. Communication interface 712 is also coupled to Internet link 734. 



assigned by the user to exchange messages with the server system. 

User interface 711 includes a keyboard and mouse. Video equipment 718 
includes a camera and monitor. Communication interface 712 includes a USB or 
serial port and a Digital Subscriber Line (DSL) modem or some other broadband 
access system. Processing system 713 includes a computer microprocessor and other 
circuitry. Storage system 714 includes a hard disk drive and other circuitry. 
Processing system 713 retrieves and executes operating software 716 and video 
software 717 from storage system 714. Software 716-717 could comprise an 
application program, firmware, or some other form of machine-readable processing 
instructions. Operating software 716 includes an operating system, networking 
software, and other utilities typically loaded onto a personal computer. When 
executed by processing system 713, video software 717 directs processing system 713 
to operate in accord with the invention. 



from the telephone line, battery, AC connection, or another source. RJ-1 1 jacks 721- 
722, DTMF decoder 723, call sensor 724, and computer interface 726 could be 
conventional components. Controller 725 comprises processing circuitry configured 
to operate in accord with the invention. Interface device 704 can be turned on and off 



to control video call initiation. 

In operation, operating software 716 directs processing system to retrieve 
and execute video software 717 in response to computer start-up or user input. Video 
software 717 directs processing system 713 to operate as follows. Processing system 
713 maintains a set of user options that can be viewed and altered through user 
interface 711. A table of possible options follows. 



Computer system 701 uses Transaction Control Protocol port 80 or any other port 



Interface device 704 could be a stand-alone enclosure that derives power 
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Task 


Options 


Log-in 


Automatic at system start, upon user input, or upon 
incoming/outgoing telephone call. 


Video call initiation 


1 . On or off. 

2. If on, then automatic, automatic using a positive or 
negative telephone number list, or after user prompt and 
acceptance. 

3. Bi-directional or unidirectional, and if unidirectional, 
then caller to called party or called party to caller. 


Video call reception 


1 . On or off. 

2. If on, then automatic, automatic using a positive or 
negative telephone number list, or after user prompt and 
acceptance. 

3. Bi-directional or unidirectional, and if unidirectional, 
then caller to called party or called party to caller. 


Encryption 


1 . On or off. * 

2. If on, then automatic, automatic using a positive or 
negative telephone number list, or after user prompt and 
acceptance. 


Video quality 


1 . Various standards that can be used based video 
transfer or receipt, positive or negative telephone number 
list, or after user prompt and selection. 



Processing system 713 generates and transfers a log-in message through 
communication interface and over the public data network to the server system. The 
login message include user name and password, user telephone number and data 
5 address, and video call parameters. 1 

Telephone 703 is operated to transfer DTMF digits to the public telephone 
network. The public telephone network processes the DTMF digits to extend the call 
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to the called party. I^^F decoder 723 monitors the telepho^^onnection between 
jacks 721-722 to detect and decode any DTMF tones transmitted by telephone 703 to 
the public telephone network. DTMF decoder 723 indicates the decoded digits to 
controller 725. Controller 725 forms the called number from the decoded digits and 
5 transfers a telephone call initiation message through computer interface 726 and USB 
connection 733. 

Processing system 713 receives the telephone call initiation message from 
communication interface 712. Processing system 713 implements the video call 
initiation options, and if the video call should proceed, processing system 713 
10 generates a video call request including the called telephone number, user name and 
password, user data address and telephone number, and video call parameters. 
Processing system 713 transfers the video call request message through 
communication interface 712 to Internet link 734 for delivery to the server system 
over the Internet. Processing system 713 then awaits a video call start message from 
15 the server system with the parameters for the video call. When the video call start 

message is received, processing system implements the video call parameters and user 
j options. For a bidirectional video call, processing system 713 directs the exchange of 

video between video equipment 718 and Internet link 734. Video equipment 718 
! displays the video. 

bo Eventually, telephone 703 is placed on-hook. Call sensor 724 monitors the 

telephone connection between jacks 721-722 to detect the on-hook condition. 
Typically, call sensor 724 monitors line current to detect off-hook and on-hook 
conditions. Call sensor 724 indicates the on-hook condition to controller 725. 
Controller 725 transfers a telephone call termination message through computer 

25 interface 726 and USB connection 733. 

Processing system 713 receives the telephone call termination message 
through communication interface 712. In response, processing system 713 generates 
and transfers a video call termination message through communication interface 712 
to Internet link 734 for delivery to the server system over the Internet. Processing 

30 system 713 directs video equipment 718 to.stop the generation and display of video. 
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It should ^appreciated that the user may operatelheir telephone in the 
normal manner and corresponding video calls are automatically established over the 
Internet. The telephone calls provides the audio, and the Internet connection transfers 
the video. As indicated, options are available to exert various levels of user control 
over the process. 

To receive a video call, processing system 713 receives a video call request 
from the server system over Internet link 734 and through communication interface 
712. Processing system 713 implements any user options and may notify the user 
through user interface 71 1 or video equipment 718. If the video call is accepted, 
processing system 713 transfers a video call acceptance to the server system through 
communication interface 712 and over Internet link 734. 

When the video call start message is received, processing system 713 
implements the video call parameters and user options. For a bidirectional video call, 
processing system 713 directs the exchange of video between video equipment 718 
and Internet link 734. Video equipment 718 displays the video. Video call 
termination may proceed as indicated above or a video call termination message may 
be received from the server system. If a video call termination message is received, 
then processing system 713 directs video equipment 718 to stop the generation and 

* * ! 1 

display of video. 

It should be appreciated that the user may invoke video software 717 to 
dynamically control video calls. For example, video software 717 may allow the user 
to terminate video calls in one or both directions during the call. Video software 717 
may allow the user to adjust user options during the call. Video software 717 may 
allow the user to initiate a previously rejected video call during the telephone call. 
Computer system 701, telephone 703, and interface device 704 can be configured to 
operate together for additional user control. In such a scenario, the user could transfer 
DTMF digits that are decoded by interface device 704 and transferred to computer 
system 701 to exert control. For example, incoming video calls could be accepted or 
rejected by transferring specific DTMF sequences from telephone 703. Video calls 
could be terminated by the user in a similar fashion. 
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Interface device 704 could be further equipped with a tone generator to alert 
the user to various conditions. For example, interface device 704 could transfer a 
special tone to telephone link 73 1 for the user to hear to indicate that a video call is 
available for the telephone call. The tone could be played in response to a video call 
request or start message from the server system. 

Server System - Figures 16-18 

Figure 16 illustrates server system 800 in an example of the invention. Server system 
800 includes user interface 801, network interface 802, processing system 803, and 
storage system 804. Storage system 804 stores operating software 806 and video 
software 807. Network interface 802 is coupled to Internet connection 817. 
Processing system 803 uses network interface 802 to communicate over the Internet 
with user systems. 

Processing system 803 retrieves and executes operating software 806 and 
video software 807 from storage system 804. Software 806-807 could comprise an 
application program, firmware, or some other form of machine-readable processing 
instructions. Operating software 806 includes an operating system, networking 
software, and other utilities typically loaded onto an Internet server. When executed 
by processing system 803, video software 807 directs processing system 803 to control 
server system 800 in accord with the invention. 

Figures 17-18 illustrate server system 800 operation in an example of the 
invention. Server system 800 maintains a database of users including user names and 
passwords, user telephone numbers and data addresses, and possibly user preferences. 
If server system 800 receives a log-in message, the user password is checked and if it 
is valid, the user database is modified to indicate that the user is ready to initiate 
and/or receive video calls. If server system 800 receives a log-off message, then the 
user password is checked and if it is valid, the user database is modified to indicate 
that the user is not ready to initiate and/or receive video calls. The user database may 
also be modified by querying the users and receiving responses indicating user video 
call specifications. 
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If server system 800 receives a video -call request, tne user database is 
checked using the called telephone number from the request to determine if the called 
party is ready to receive video calls. If not, a video call unavailable message is 
returned to the caller. If so, server system 800 sends a video call request to the called 
party. If a video call rejection is received from the called party, then server system 
800 sends a video call rejection to the caller. If a video call acceptance is received 
from the called party, server system resolves video call parameters and sends video 
call start messages to both the caller and called party including the video call 
parameters. Parameter resolution may entail determining if the caller will receive 
called party video. If so, this is indicated in the video start messages. 

Server system 800 uses the called party telephone number to retrieve the 
called party data address. Server system 800 may use the caller telephone number to 
retrieve the caller data address. If server system 800 receives video from the caller, it 
addresses the caller video to the called party data address and transfers the caller video 
for delivery to the called party. If server system 800 receives video from the called 
party, it addresses the called party video to the caller data address and transfers the 
called party video for delivery to the caller. In some cases, it may be necessary for 
processing system 803 to interwork the video so it is compatible with both caller and 
called party. For example, the caller and called party may use different quality or 
encryption levels that are interworked by processing system 803. 

If server system 800 receives a video call termination message from one 
user, it transfers a video call termination message to the other user. Server system 800 
then modifies the database to return each user to their pre-call status. Server system 
also generates billing information for the video call. 

Server system 800 may be configured to download software to the user 
systems. The software could be the video software used to control the user systems as 
described above. The software could provide upgrades from older versions. The 
software could provide video processing, compression, and encryption. The software 
could provide system diagnostics and trouble-shooting to recommend optimal system 
software and settings. 
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It should ^Appreciated that the processing and c^Aol discussed above 
could be distributed in various ways between the user system and the server system. 
For example, the server system could maintain and implement user preferences and 
provide user prompts. 
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